Papers by Jake W. Vincent
MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization (2025.findings-emnlp)
Copied to clipboard
Yinhong Liu, Jianfeng He, Hang Su, Ruixue Lian, Yi Nian, Jake W. Vincent, Srikanth Vishnubhotla, Robinson Piramuthu, Saab Mansour
| Challenge: | Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. |
| Approach: | They propose a meta-evaluation benchmark for multimodal dialogue summarization based on image-sharing dialogues, corresponding summaries and human judgments . |
| Outcome: | The proposed framework is the first to identify and formalize key evaluation dimensions specific to MDS. |
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (2025.naacl-long)
Copied to clipboard
Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, Hang Su
| Challenge: | Existing approaches to evaluate faithfulness of summaries are often fooled by the fluency of the text and struggle with identifying errors. |
| Approach: | They propose an approach to summary faithfulness evaluation where multiple LLM-based agents are assigned initial stances and forced to come up with a reason to justify belief. |
| Outcome: | The proposed approach can identify ambiguities and have even stronger performance on non-ambiguous summaries. |